A View From Egnyte Engineering

May 31, 2011

Filers, Raid6, & IOPs

Krishna Sankar

I am Krishna Sankar and I recently joined the Egnyte team. While not my first tour of duty, our favorite community officer, Emily, asked me to contribute to the blog about Egnyte Labs. We love our work on scalability, web infrastructure, and of course storage hardware and software layers. We will write about the topics we are wrestling with and the pointers we encountered during our research and our solutions.

This week, some of our musings on high capacity storage systems.

Our filer hardware is JBOD with Raid6 and a few other characteristics. So far our philosophy has been the bigger the better -- chassis with 16+ disk arrays and with 2 TB, each reaching 32+ TB!

Now we are working on our next generation object store, which can scale at least 10 times, is more fault tolerant and resilient, offers a richer granular feature set and gives us more IOPs. As we design the new system we are realizing that many things do not work at 10X scale, optimization at 10X scale is different, and the laws of physics do change at 10X scale. Interesting problems to have and solve.

For example, Raid 6. We do not want Raid6. In our view the most vulnerable system in a filer is the hard disk. With Raid6 and hot spare we are covered for normal wear and tear at the disk level. But Raid6 does take some IOPs and as we scale, we are exploring ways of recovering some of the IOPs back.
Our solution is to use smaller filers and more of them. We are exploring limiting our filers to about 16 TB. We want our hardware and OS layer to be as simple as possible, making it easier for ops at that layer. Having smaller filers gives us some IOPs back and recovery (if we need it) is faster.
Another trick to speedup is to have SRAM in the Raid6 cards and dedicate the entire SRAM as write-through. Every little trick helps, especially at scale.

We welcome your ideas and would like to hear about your experiences with scalable performance at the storage hardware and OS layer without giving up operational simplicity.

Next blog...an adventure into the land of object stores.